LARGE VISION-LANGUAGE MODELS

Enhancing Adversarial Transferability in Visual-Language Pre-training Models via Local Shuffle and Sample-based Attack (2025.findings-naacl)

Challenge:	Recent research has focused on addressing multimodal hallucinations in Large Vision-Language Models (LVLMs) however, these methods lack fine-grained visual contrast mechanisms and rely on single-margin optimization.
Approach:	They propose a framework that integrates text-conditioned preference loss with visual ranking-based objective.
Outcome:	The proposed framework improves cross-modal alignment and fine-grained visual grounding.

Copied to clipboard

Xin Liu, Aoyang Zhou, Kun He

LOTUS: A Leaderboard for Detailed Image Captioning from Quality to Societal Bias and User Preferences (2025.acl-industry)

Challenge:	Visual-Language Pre-training (VLP) models are vulnerable to adversarial examples . previous studies have focused on improving adversariality of models .
Approach:	They propose a local shuffle and sample-based attack that randomly shufts one of the local image blocks and generates adversarial images and samples around them.
Outcome:	The proposed attack outperforms other advanced attacks on Large Vision-Language Models and outperformed previous attacks on Visual-Langue Pre-training models.

Copied to clipboard

Yusuke Hirota, Boyi Li, Ryo Hachiuma, Yueh-Hua Wu, Boris Ivanovic, Marco Pavone, Yejin Choi, Yu-Chiang Frank Wang, Yuta Nakashima, Chao-Han Huck Yang

Mind Your Special Tokens! On the Importance of Dedicated Sequence-End Tokens in Vision-Language Embedding Models (2026.eacl-short)

Challenge:	Large Vision-Language Models (LVLMs) have transformed image captioning . existing evaluations lack standardized criteria and a standardized evaluation framework .
Approach:	They propose a leaderboard for evaluating detailed captions that addresses three main gaps in existing evaluations: lack of standardized criteria, bias-aware assessments, and user preference considerations.
Outcome:	The proposed model evaluates caption quality, descriptiveness, risks, and societal biases while tailoring criteria to user preferences.

Copied to clipboard

Elio Musacchio, Giovanni Semeraro, Goran Glavaš

Through the Looking Glass: Common Sense Consistency Evaluation of Weird Images (2025.naacl-srw)

Challenge:	Large Vision-Language Models (LVLMs) are highly sensitive to end-of-input artifacts in fine-tuning and inference data, e.g., whether input sequences end with punctuation or newline characters.
Approach:	They propose to convert generative LVLMs into vision-language encoders via contrastive learning objectives and use supervised contrastive objectives to train them.
Outcome:	The proposed approach improves visual and text representations and improves retrieval and (semantic) similarity tasks.

Copied to clipboard

Elisei Rykov, Kseniia Petrushina, Kseniia Titova, Anton Razzhigaev, Alexander Panchenko, Vasily Konovalov

PatentVision: A multimodal method for drafting patent applications (2026.eacl-industry)

Challenge:	Existing methods to measure image common sense inconsistentness are difficult to implement because of their complexity.
Approach:	They propose a visual commonsense model that leverages large vision-language models to extract atomic facts from images and a compact attention-pooling classifier to fine-tune it over encoded atomic fact.
Outcome:	The proposed method outperforms existing methods on the WHOOPS! and WEIRD datasets while maintaining a compact attention-pooling classifier over encoded atomic facts.

Copied to clipboard

Ruo Yang, Sai Krishna Reddy Mudhiganti, Manali Sharma

EmbSpatial-Bench: Benchmarking Spatial Understanding for Embodied Tasks with Large Vision-Language Models (2024.acl-short)

Challenge:	PatentVision integrates textual and visual inputs to generate patent specifications . existing systems fail to capture the nuanced interplay between textual, visual components .
Approach:	They propose a multimodal framework that integrates textual and visual inputs to generate patent specifications.
Outcome:	The proposed framework surpasses text-only methods in patent writing, the authors show . it integrates visual data to better represent intricate design features and functional connections .

Copied to clipboard

Mengfei Du, Binhao Wu, Zejun Li, Xuanjing Huang, Zhongyu Wei

VGA: Vision GUI Assistant - Minimizing Hallucinations through Image-Centric Fine-Tuning (2024.findings-emnlp)

Challenge:	Recent studies have revealed significant deficiencies of LVLMs in understanding visual contents, leaving the gap between current embodied intelligence and large vision-language models (LVLM) .
Approach:	They propose to use a benchmark to evaluate LVLMs' spatial understanding of embodied environments to evaluate their ability to understand visual contents.
Outcome:	The proposed benchmark is derived from embodied scenes and covers 6 spatial relationships from an egocentric perspective.

Copied to clipboard

Meng Ziyang, Yu Dai, Zezheng Gong, Shaoxiong Guo, Minglong Tang, Tongquan Wei

Jailbreak Large Vision-Language Models Through Multi-Modal Linkage (2025.acl-long)

Challenge:	Existing Large Vision-Language Models (VLMs) often overly rely on internal text-based knowledge while neglecting visual inputs.
Approach:	They propose a model that balances attention image and text to enhance interpretation and reduce hallucinations by using a visual input.
Outcome:	The proposed model improves interpretation and reduces hallucinations by balancing attention image and text to enhance interpretation and reduction of hallucinosity.

Copied to clipboard

Yu Wang, Xiaofei Zhou, Yichen Wang, Geyuan Zhang, Tianxing He

DRUM: Learning Demonstration Retriever for Large MUlti-modal Models (2025.acl-srw)

Challenge:	Existing methods to jailbreak large vision-language models fail against cutting-edge models such as GPT-4o, despite having undergone safety alignment training.
Approach:	They propose a new framework for jailbreaking large vision-language models that uses an encryption-decryption process to mitigate the over-exposure of harmful information.
Outcome:	The proposed framework jailbreaks GPT-4o with 99.40% success rates on SafeBench, 98.81% on MM-SafeBench and 99.07% on HADES-Dataset.

Copied to clipboard

Ellen Yi-Ge, Jiechao Gao, Wei Han, Wei Zhu

Mitigating Hallucinations in Large Vision-Language Models without Performance Degradation (2026.acl-long)

Challenge:	Recent studies have shown that large language models (LLMs) have impressive capabilities in dealing with new tasks with the help of in-context learning (ICL).
Approach:	They propose to concate the image and text embeddings to enhance the retrieval performance of a visual-language task and to calculate a list-wise ranking loss for training the embeddable model.
Outcome:	The proposed framework fine-tunes the CLIP embedding model to better meet the needs of the large vision-language models.

Copied to clipboard

Xingyu Zhu, Junfeng Fang, Shuo Wang, Beier Zhu, Zhicai Wang, Yonghui Yang, Xiangnan He

Large Vision-Language Model Alignment and Misalignment: A Survey Through the Lens of Explainability (2025.findings-emnlp)

Challenge:	Recent advances in large vision-language models produce hallucinations that compromise output reliability.
Approach:	They propose a dual-stage framework for mitigating hallucinations without performance degradation . they propose semantic-aware component disentanglement and interpretable parameter updates .
Outcome:	The proposed model reduces hallucinations by 23.4% while maintaining 97.4% of general generative capability.

Copied to clipboard

Dong Shu, Haiyan Zhao, Jingyu Hu, Weiru Liu, Ali Payani, Lu Cheng, Mengnan Du

Explainability & Interpretability in NLP

Challenge:	Large Vision-Language Models have demonstrated remarkable capabilities in processing both visual and textual information.
Approach:	They examine the challenge of alignment and misalignment in LVLMs through an explainability lens.
Outcome:	The findings highlight the need for standardized evaluation protocols and in-depth explainability studies.

Evaluating Vision-Language Models for Emotion Recognition (2025.findings-naacl)

Copied to clipboard

Sree Bhattacharyya, James Z. Wang

HELPD: Mitigating Hallucination of LVLMs by Hierarchical Feedback Learning with Vision-enhanced Penalty Decoding (2024.emnlp-main)

Challenge:	Large Vision-Language Models (VLMs) have been used for objective multimodal reasoning tasks for decades.
Approach:	They present a comprehensive evaluation of large vision-language models for recognizing evoked emotions from images.
Outcome:	The proposed model performs well in evoked emotion recognition task and is robust to human errors.

Copied to clipboard

Fan Yuan, Chi Qin, Xiaogang Xu, Piji Li

VALOR-EVAL: Holistic Coverage and Faithfulness Evaluation of Large Vision-Language Models (2024.findings-acl)

Challenge:	Existing work detects hallucination by directly judging whether an object exists in an image, overlooking the association between the object and semantics.
Approach:	They propose a framework that incorporates hallucination feedback at both object and sentence semantic levels to alleviate over 15% of hallucinism.
Outcome:	The proposed framework can alleviate over 15% of hallucination even with a marginal degree of training.

Copied to clipboard

Haoyi Qiu, Wenbo Hu, Zi-Yi Dou, Nanyun Peng

Deploying Tiny LVLM Judges for Real-World Evaluation of Chart Models: Lessons Learned and Best Practices (2025.emnlp-industry)

Challenge:	Existing evaluation methods focus on object hallucinations, focusing on object outputs . current evaluation methods struggle to address subtle semantic distinctions between outputs and reference data .
Approach:	They propose a multi-dimensional benchmark covering objects, attributes, and relations . they propose metric that generalizes CHAIR metric and incorporates faithfulness and coverage .
Outcome:	The proposed evaluation framework is more comprehensive and better correlated with humans than existing evaluation methods.

Copied to clipboard

Md Tahmid Rahman Laskar, Mohammed Saidul Islam, Ridwan Mahbub, Mizanur Rahman, Amran Bhuiyan, Israt Jahan, Mir Tafseer Nayeem, Shafiq Joty, Enamul Hoque, Jimmy Huang

Centurio: On Drivers of Multilingual Ability of Large Vision-Language Model (2025.acl-long)

Challenge:	Large Vision-Language Models (LVLMs) with only 7B parameters perform poorly as judges in resource-constrained settings.
Approach:	They propose two approaches to ensure costefficient evaluation by combining multiple criteria into a single query and domainadaptive transfer learning to create a 2Bparameter VLM on a chart dataset.
Outcome:	The proposed model can effectively transfer knowledge from one dataset to another to make it a more specialized model.

Copied to clipboard

Gregor Geigle, Florian Schneider, Carolin Holtermann, Chris Biemann, Radu Timofte, Anne Lauscher, Goran Glavaš

African or European Swallow? Benchmarking Large Vision-Language Models for Fine-Grained Object Classification (2024.emnlp-main)

Challenge:	Existing models for large vision-language tasks are trained on English data, which makes them struggle to understand non-English input and fail to generate output in the desired target language.
Approach:	They conduct multi-stage experiments on 13 vision-language tasks and 43 languages . they find that one can include as many as 100 training languages simultaneously with as little as 25-50% of non-English data .
Outcome:	The proposed model outperforms existing models in 14 tasks and 56 languages.

Copied to clipboard

Gregor Geigle, Radu Timofte, Goran Glavaš

Information Retrieval

Challenge:	Recent Large Vision Language Models demonstrate impressive abilities on image understanding and reasoning tasks.
Approach:	They propose a benchmark for fine-grained object classification that is difficult to evaluate . they benchmark 12 public LVLMs on and show CLIP models exhibit better performance .
Outcome:	The proposed model improves on 12 public LVLMs on image understanding and reasoning tasks.

Generating Spatial Knowledge Graphs from Automotive Diagrams for Question Answering (2025.emnlp-industry)

Copied to clipboard

Steve Bakos, Chen Xing, Heidar Davoudi, Aijun An, Ron DiCarlantonio

VReST: Enhancing Reasoning in Large Vision-Language Models through Tree Search and Self-Reward Mechanism (2025.acl-long)

Challenge:	Useful answers require obvious landmarks as a reference point . a decomposed pipeline is the most effective strategy for generating a high-quality SKG .
Approach:	They propose to generate a spatial knowledge graph from a vehicle dashboard diagram . they use large vision-language models to generate the graph using a decomposed pipeline .
Outcome:	The proposed method identifies landmarks with 71.3% agreement with human annotators on a new vehicle dataset.

Copied to clipboard

Congzhi Zhang, Jiawei Peng, Zhenglin Wang, Yilong Lai, Haowen Sun, Heng Chang, Fei Ma, Weijiang Yu

Visual Evidence Prompting Mitigates Hallucinations in Large Vision-Language Models (2025.acl-long)

Challenge:	Large Vision-Language Models (LVLMs) have shown exceptional performance in multimodal tasks, but their effectiveness in complex visual reasoning is constrained.
Approach:	They propose a training-free approach that enhances Reasoning in Large Vision-Language Models . they propose integrating Monte Carlo Tree Search and Self-Reward mechanisms into the reasoning tree .
Outcome:	The proposed approach surpasses current prompting methods and secures state-of-the-art performance across three multimodal reasoning benchmarks.

Copied to clipboard

Wei Li, Zhen Huang, Houqiang Li, Le Lu, Yang Lu, Xinmei Tian, Xu Shen, Jieping Ye

TABED: Test-Time Adaptive Ensemble Drafting for Robust Speculative Decoding in LVLMs (2026.findings-eacl)

Challenge:	LVLMs have shown impressive progress by integrating visual perception with linguistic understanding to produce contextually grounded outputs.
Approach:	They propose a visual evidence prompting method to mitigate hallucinations in large vision-language models by using small visual models to complement them.
Outcome:	The proposed method reduces hallucinations by reducing false activation and enhancing correct ones.

Copied to clipboard

Minjae Lee, Wonjun Kang, Byeongkeun Ahn, Christian Classen, Kevin Galim, Seunghyuk Oh, Minghao Yan, Hyung Il Koo, Kangwook Lee

Addressing Overthinking in Large Vision-Language Models via Gated Perception-Reasoning Optimization (2026.findings-acl)

Challenge:	Large Vision Language Models (LVLMs) are advanced models that process multiple modalities, such as images, audio, and video, alongside text.
Approach:	They propose to use a method to generate and verify draft tokens in parallel . they compare existing methods with small draft models and observe performance fluctuations .
Outcome:	The proposed method achieves an average walltime speedup of 1.74 over autoregressive decoding and a 5% improvement over single drafting methods.

Copied to clipboard

Xingjian Diao, Zheyuan Liu, Chunhui Zhang, Weiyi Wu, Keyi Kong, Lin Shi, Kaize Ding, Soroush Vosoughi, Jiang Gui

VLind-Bench: Measuring Language Priors in Large Vision-Language Models (2025.findings-naacl)

Challenge:	Prior work has attempted to mitigate this issue by using adaptive reasoning strategies, but these methods overlook a fundamental bottleneck: visual perception failures.
Approach:	They propose a meta-reasoning controller that dynamically routes computation among three decision paths at each generation step.
Outcome:	The proposed method outperforms slow-thinking methods while producing shorter responses.

Copied to clipboard

Kang-il Lee, Minbeom Kim, Seunghyun Yoon, Minsung Kim, Dongryeol Lee, Hyukhun Koh, Kyomin Jung

Mitigating Hallucinations in Large Vision-Language Models via Summary-Guided Decoding (2025.findings-naacl)

Challenge:	Large Vision-Language Models suffer from a problem known as language prior . such language priors can lead to undesirable biases and hallucinations when dealing with images that are out of distribution.
Approach:	They propose a benchmark to measure the language priors of Large Vision-Language Models.
Outcome:	The proposed benchmark is the first specifically designed to measure the language priors, or blindness, of LVLMs.

Copied to clipboard

Kyungmin Min, Minbeom Kim, Kang-il Lee, Dongryeol Lee, Kyomin Jung

Crafting Adversarial Inputs for Large Vision-Language Models Using Black-Box Optimization (2026.findings-eacl)

Challenge:	Large Vision-Language Models (LVLMs) generate detailed and coherent responses from visual inputs but are prone to generate hallucinations due to an over-reliance on language priors.
Approach:	They propose a method that reduces the text context and controls only the image-related POS tokens to maintain text quality by reducing the text contextualization.
Outcome:	The proposed method achieves state-of-the-art performance on object hallucination benchmarks and achieves Pareto optimality among the existing methods.

Copied to clipboard

Jiwei Guan, Haibo Jin, Haohan Wang

G-Cap: A Game Character Caption Generator (2026.acl-long)

Challenge:	Existing white-box jailbreak methods require full model accessibility and require computational costs.
Approach:	They propose a black-box jailbreak attack using Zeroth-Order optimization using ZO-SPSA.
Outcome:	The proposed method achieves highest jailbreak success rate on three LVLMs, including InstructBLIP, LLaVA and MiniGPT-4.

Copied to clipboard

Yang Yang, Feng Hu, Haiming Zhang, XU Cheng, Gui Zheng, Liang Yao, Wenqi Ren

Text Generation

Challenge:	Existing studies on Large Vision-Language Models (LVLMs) primarily focus on real-world scenarios, leaving surreal, highly stylized, and semantically hybrid virtual-world situations significantly underexplored.
Approach:	They propose to use a manually annotated benchmark to evaluate LVLMs' ability to perceive and describe game character from the virtual-world.
Outcome:	The proposed task evaluates LVLMs’ ability to perceive and describe game character from the virtual-world.

Ghost in the Shell: Synonym-Aware Logit Shaping Fingerprint for Copyright Protection of Large Vision-Language Models (2026.findings-acl)

Copied to clipboard

Xiaofan Zheng, Xinghao Wang, Xiaojun Wan

Concept-skill Transferability-based Data Selection for Large Vision-Language Models (2024.emnlp-main)

Challenge:	Existing fingerprinting methods for large vision-language models rely on backdoors to elicit abnormal outputs, but direct distortion of the model’s original outputs compromises modality alignment and degrades multimodal capabilities.
Approach:	They propose to embed a robust fingerprint while preserving the original normal outputs of the model.
Outcome:	The proposed fingerprint maintains multimodal performance and substantially enhances fingerprint robustness.

Copied to clipboard

Jaewoo Lee, Boyang Li, Sung Ju Hwang

Look, Compare, Decide: Alleviating Hallucination in Large Vision-Language Models via Multi-View Multi-Path Reasoning (2025.coling-main)

Challenge:	Large Vision-Language Models (LVLMs) require instruction tuning on extensive data . training on large VL datasets can be prohibitively expensive .
Approach:	They propose a data selection technique that uses a small model as a reference model to select training data for efficient finetuning of a target LVLM.
Outcome:	The proposed method achieves superior performance and data selection efficiency against 8 strong baselines on two distinct datasets: LLaVA-1.5 and Vision-Flan.

Copied to clipboard

Xiaoye Qu, Jiashuo Sun, Wei Wei, Daizong Liu, Jianfeng Dong, Yu Cheng

Uncovering Bias in Large Vision-Language Models at Scale with Counterfactuals (2025.naacl-long)

Challenge:	Large Vision-Language Models (LVLMs) have impressive capabilities in multi-modal context comprehension, but they still suffer from hallucination problems due to inconsistent outputs with the image content.
Approach:	They propose a training-free framework MVP to reduce hallucinations in Large Vision-Language Models . they propose multi-view information-seeking strategy to perceive the comprehensive information in the image .
Outcome:	The proposed framework reduces hallucinations in large vision-language models by combining multi-view multi-path reasoning with multi-vision multi-path reasoning.

Copied to clipboard

Phillip Howard, Kathleen C. Fraser, Anahita Bhiwandiwalla, Svetlana Kiritchenko

Evian: Towards Explainable Visual Instruction-tuning Data Auditing (2026.findings-acl)

Challenge:	Large Vision-Language Models (LVLMs) have been proposed to augment LLMs with visual inputs.
Approach:	They propose large vision-Language Models to augment LLMs with visual inputs.
Outcome:	The proposed models condition generated text on both an input image and a visual prompt, enabling a variety of use cases such as visual question answering and multimodal chat.

Copied to clipboard

Zimu Jia, Mingjie Xu, Andrew Estornell, Jiaheng Wei

A Cognitive Evaluation Benchmark of Image Reasoning and Description for Large Vision-Language Models (2025.naacl-long)

Challenge:	Existing data filtering methods rely on coarse-grained scores that lack granularity to identify nuanced semantic flaws.
Approach:	They propose a "Decomposition-then-Evaluation" paradigm that breaks model responses into constituent cognitive components.
Outcome:	The proposed model outperforms models trained on larger datasets in three key areas . the authors show that Logical Coherence is the most critical factor in data quality evaluation .

Copied to clipboard

Xiujie Song, Mengyue Wu, Kenny Q. Zhu, Chunhao Zhang, Yanyi Chen

Reducing Token Redundancy in LVLMs: A Systematic Review of Token Pruning Methods (2026.acl-long)

Challenge:	Large Vision-Language Models (LVLMs) are hardly comprehensively evaluated for their cognitive abilities.
Approach:	They propose to evaluate high-level cognitive abilities of Large Vision-Language Models (LVLMs) using images with rich semantics.
Outcome:	The proposed evaluation benchmark consists of 251 images along with comprehensive annotations.

Copied to clipboard

Hanzhang Yuan, Mengxuan Hu, Wenhao Zhang, Tianlong Wang, Zhongliang Zhou, Jiasen Lu, Sheng Li

Evaluating and Mitigating Object Hallucination in Large Vision-Language Models: Can They Still See Removed Objects? (2025.naacl-long)

Challenge:	Large Vision-Language Models (LVLMs) excel at visual understanding but face severe computational bottlenecks when processing high-resolution images and long videos due to massive visual token counts.
Approach:	They propose a taxonomy categorizing methods into vision-side, LLM-side and hybrid paradigms and analyze token selection mechanisms and pruning strategy.
Outcome:	The proposed method selectively removes less informative tokens while maintaining performance.

Copied to clipboard

Yixiao He, Haifeng Sun, Pengfei Ren, Jingyu Wang, Huazheng Wang, Qi Qi, Zirui Zhuang, Jing Wang

Mitigating Hallucinations in Large Vision-Language Models (LVLMs) via Language-Contrastive Decoding (LCD) (2024.findings-acl)

Challenge:	LVLMs often mistakenly determine objects as present in images where they do not exist . authors propose a new benchmark to evaluate object hallucinations by removing objects from images and asking the model whether it can still see the removed objects.
Approach:	They propose a benchmark to evaluate object hallucinations by removing objects from images . they propose oDPO, a direct preference optimization objective based on visual objects .
Outcome:	The proposed benchmark reduces the likelihood of object hallucinations by removing objects from images and asking the model whether it can still see the removed objects.

Copied to clipboard

Avshalom Manevich, Reut Tsarfaty

Hierarchical Safety Realignment: Lightweight Restoration of Safety in Pruned Large Vision-Language Models (2025.findings-acl)

Challenge:	Large Vision-Language Models (LVLMs) often produce object hallucinations due to their reliance on text cues and learned object co-occurrence biases.
Approach:	They propose a language-contrasting decoding algorithm that adjusts LVLM outputs based on LLM confidence levels to mitigate object hallucinations.
Outcome:	The proposed method shows up to %4 improvement in POPE F1 scores and %36 reduction in CHAIR scores on COCO validation set while improving captioning quality scores.

Copied to clipboard

Yue Li, Xin Yi, Dongsheng Shi, Gerard De Melo, Xiaoling Wang, Linlin Wang

Retrieval Visual Contrastive Decoding to Mitigate Object Hallucinations in Large Vision-Language Models (2025.findings-acl)

Challenge:	Recent work has shown that pruning can reduce model performance, but it can also lead to degradation in safety performance.
Approach:	They propose a hierarchical safety realignment approach to prune large vision-Language Models . they quantify contribution of each attention head to safety and restore neurons .
Outcome:	The proposed approach achieves significant safety improvements in LVLMs pruned post pruning.

Copied to clipboard

Jihoon Lee, Min Song

SURf: Teaching Large Vision-Language Models to Selectively Utilize Retrieved Information (2024.emnlp-main)

Challenge:	Large Vision Language Models are not free from the issue of Object Hallucination (OH) OH is a phenomenon where LVLMs generate hallucinated objects and descriptions in their outputs.
Approach:	They propose a method to suppress OH by referencing images from AI-generated images at the logit level.
Outcome:	The proposed method significantly improves existing methods on visual contrast decoding.

Copied to clipboard

Jiashuo Sun, Jihai Zhang, Yucheng Zhou, Zhaochen Su, Xiaoye Qu, Yu Cheng

DAMRO: Dive into the Attention Mechanism of LVLM to Reduce Object Hallucination (2024.emnlp-main)

Challenge:	Existing studies focus on the text modality or are limited to specific tasks.
Approach:	They propose a framework to teach Large Vision-Language Models to selectively utilize retrieved information and improve their robustness against irrelevant or misleading references.
Outcome:	The proposed framework improves LVLMs’ ability to utilize retrieved multimodal references and their robustness against irrelevant or misleading information.

Copied to clipboard

Xuan Gong, Tianshi Ming, Xinpeng Wang, Zhihua Wei

Watermarking for Factuality: Guiding Vision-Language Models Toward Truth via Tri-layer Contrastive Decoding (2025.findings-emnlp)

Challenge:	Despite the success of Large Vision-Language Models, they suffer from hallucination.
Approach:	They propose a training-free strategy that "Dive into" the attention of LVLMs to "Reduce" object hallucination by using classification tokens of ViT.
Outcome:	The proposed method reduces the impact of outlier tokens on LVLMs . the proposed method is based on LLaVA-1.5, LLvaVA-NeXT and InstructBLIP .

Copied to clipboard

Kyungryul Back, Seongbeom Park, Milim Kim, Mincheol Kwon, SangHyeok Lee, Hyunyoung Lee, Junhee Cho, Seunghyun Park, Jinkyu Kim

MIND: Multimodal Shopping Intention Distillation from Large Vision-language Models for E-commerce Purchase Understanding (2024.emnlp-main)

Challenge:	Large Vision-Language Models (LVLMs) have shown promising results on multimodal tasks, but remain prone to hallucinations due to their reliance on a single modality or memorizing training data without properly grounding their outputs.
Approach:	They propose a training-free, tri-layer contrastive decoding with watermarking that uses a watermark-related question to identify a pivot layer and apply tri-layered contrastive coding to generate the final output.
Outcome:	The proposed method reduces hallucinations and generates more visually grounded responses.

Copied to clipboard

Baixuan Xu, Weiqi Wang, Haochen Shi, Wenxuan Ding, Huihao Jing, Tianqing Fang, Jiaxin Bai, Xin Liu, Changlong Yu, Zheng Li, Chen Luo, Qingyu Yin, Bing Yin, Long Chen, Yangqiu Song

Green & Sustainable NLP

Challenge:	Existing methods for acquiring large-scale intentions generate product-centric intentions without product images and incur high costs for scalability.
Approach:	They propose a multimodal framework that allows Large Vision-Language Models to infer purchase intentions from multimodal product metadata and prioritize human-centric ones.
Outcome:	The proposed framework shows that it is robust to different prompts and superior to previous methods.

Mixture of Decoding: An Attention-Inspired Adaptive Decoding Strategy to Mitigate Hallucinations in Large Vision-Language Models (2025.findings-acl)

Copied to clipboard

Xinlong Chen, Yuanxing Zhang, Qiang Liu, Junfei Wu, Fuzheng Zhang, Tieniu Tan

K-GIP: Diagnosing Logical Fractures in Large Vision-Language Models via Verification Scene Graphs and Sequential Pruning (2026.findings-acl)

Challenge:	Large Vision-Language Models (LVLMs) have impressive capabilities across visual tasks, yet they remain hindered by the persistent challenge of hallucinations.
Approach:	They propose a novel approach that dynamically adapts decoding strategies by evaluating the correctness of the model’s attention on image tokens to distinguish the correct attention.
Outcome:	Extensive experiments show that the proposed approach outperforms existing decoding methods across multiple mainstream benchmarks, effectively mitigating hallucinations in LVLMs.

Copied to clipboard

Yujun Hu, Xiaoyu Zhou, Changbo Wang, Gaoqi He

GIMMICK: Globally Inclusive Multimodal Multitask Cultural Knowledge Benchmarking (2025.findings-acl)

Challenge:	Existing benchmarks that treat hallucinations as isolated errors neglect causal dependencies between visual perception and textual reasoning.
Approach:	They propose a Knowledge-Guided In-Context Probing framework that constructs a dual-perception ground truth to transform abstract priors into multi-granularity queries.
Outcome:	The proposed framework isolates deep reasoning failures from simple perceptual misses.

Copied to clipboard

Florian Schneider, Carolin Holtermann, Chris Biemann, Anne Lauscher

DivScene: Towards Open-Vocabulary Object Navigation with Large Vision Language Models in Diverse Scenes (2025.findings-emnlp)

Challenge:	Existing studies on Large Vision-Language Models (LVLMs) focus on a narrow range of cultures, focus on only a small number of cultural aspects or evaluate a limited selection of models on ONE task only.
Approach:	They propose a multimodal benchmark to assess a broad spectrum of cultural knowledge across 144 countries representing six global macro-regions.
Outcome:	The proposed benchmark examines cultural knowledge across 144 countries across six global macro-regions.

Copied to clipboard

Zhaowei Wang, Hongming Zhang, Tianqing Fang, Ye Tian, Yue Yang, Kaixin Ma, Xiaoman Pan, Yangqiu Song, Dong Yu

GAOKAO-MM: A Chinese Human-Level Benchmark for Multimodal Models Evaluation (2024.findings-acl)

Challenge:	Large Vision-Language Models (LVLMs) have achieved significant progress in tasks like visual question answering and document understanding.
Approach:	They introduce DivScene, a large-scale dataset with 4,614 houses across 81 scene types and 5,707 kinds of target objects.
Outcome:	The proposed dataset provides a much greater diversity of target objects and scene types than existing datasets, enabling a comprehensive task evaluation.

Copied to clipboard

Yi Zong, Xipeng Qiu

Towards Statistical Factuality Guarantee for Large Vision-Language Models (2025.emnlp-main)

Challenge:	Existing datasets focus on primary perception abilities and commonsense knowledge, or have low level of text comprehension difficulty, which are insufficient to reflect comprehensive capabilities of large vision-language models.
Approach:	They propose a multimodal benchmark based on the Chinese College Entrance Examination (GAOKAO) which sets human-level requirements for the model’s abilities, including perception, understanding, knowledge and reasoning.
Outcome:	The proposed model derives from native Chinese context and sets human-level requirements for its abilities, including perception, understanding, knowledge and reasoning.

Copied to clipboard

Zhuohang Li, Chao Yan, Nicholas J Jackson, Wendi Cui, Bo Li, Jiaxin Zhang, Bradley A. Malin

ChartGaze: Enhancing Chart Understanding in LVLMs with Eye-Tracking Guided Attention Refinement (2025.emnlp-main)

Challenge:	Advancements in Large Vision-Language Models (LVLMs) have demonstrated impressive performance in image-conditioned text generation, but hallucinated outputs pose a major barrier to their use in safety-critical applications.
Approach:	They propose a conformal-prediction-based framework that achieves finite-sample distribution-free statistical guarantees to the factuality of LVLM output.
Outcome:	The proposed framework reduces the error rate of LLaVa-1.5 claims from 87.8% to 10.0% while ensuring that the output is accurate.

Copied to clipboard

Ali Salamatian, Amirhossein Abaskohi, Wan-Cyuan Fan, Mir Rayat Imtiaz Hossain, Leonid Sigal, Giuseppe Carenini

FIHA: Automated Fine-grained Hallucinations Evaluations in Large Vision Language Models with Davidson Scene Graphs (2025.findings-acl)

Challenge:	Chart question answering (CQA) is a key research challenge for large vision-language models . recent efforts focus on leveraging LVLMs directly on chart images .
Approach:	They propose a gaze-guided attention refinement that aligns image-text attention with human fixations to improve chart reasoning quality and interpretability.
Outcome:	The proposed approach improves answer accuracy and attention alignment yielding gains of up to 2.56 percentage points across multiple models.

Copied to clipboard

Bowen Yan, Zhengsong Zhang, Liqiang Jing, Eftekhar Hossain, Xinya Du

Typographic Attacks in a Multi-Image Setting (2025.naacl-long)

Challenge:	Current approaches to large vision-language models rely on costly annotations and are not comprehensive in terms of evaluating all aspects.
Approach:	They propose an automated method which can access LVLMs hallucination in an LLM-free and annotation-free way and model the dependency between different types of halluciNations.
Outcome:	The proposed model can model the dependency between different types of hallucinations and generate Q&A pairs on any image dataset at minimal cost.

Copied to clipboard

Xiaomeng Wang, Zhengyu Zhao, Martha Larson

Activation Steering Decoding: Mitigating Hallucination in Large Vision-Language Models through Bidirectional Hidden State Intervention (2025.acl-long)

Challenge:	Large Vision-Language Models are susceptible to typographic attacks, which are misclassifications caused by an attack text that is added to an image.
Approach:	They propose a multi-image setting for studying typographic attacks by leveraging the difficulty of the target image, the strength of the attack text, and text-image similarity.
Outcome:	The proposed approach improves success rates by 21% over random, non-specific methods on the CLIP model while maintaining stealth in a multi-image scenario.

Copied to clipboard

Jingran Su, Jingfan Chen, Hongxin Li, Yuntao Chen, Li Qing, Zhaoxiang Zhang

CLAIM: Mitigating Multilingual Object Hallucination in Large Vision-Language Models with Cross-Lingual Attention Intervention (2025.acl-long)

Challenge:	Large Vision Language Models (LVLMs) suffer from hallucination where generated textual descriptions fail to align accurately with visual semantics.
Approach:	They propose a training-free approach that mitigates hallucination through targeted intervention in the model’s intermediate activations by identifying directional patterns of hallucinism in the activation space using a small calibration set.
Outcome:	The proposed approach reduces hallucination across multiple benchmarks while maintaining performance on general visual understanding tasks.

Copied to clipboard

Zekai Ye, Qiming Li, Xiaocheng Feng, Libo Qin, Yichong Huang, Baohang Li, Kui Jiang, Yang Xiang, Zhirui Zhang, Yunfei Lu, Duyu Tang, Dandan Tu, Bing Qin

Investigating and Mitigating the Multimodal Hallucination Snowballing in Large Vision-Language Models (2024.acl-long)

Challenge:	Large Vision-Language Models (LVLMs) have impressive multimodal abilities but remain prone to multilingual object hallucination.
Approach:	They propose a cross-lingual attention intervention method to mitigate multilingual object hallucination in LVLMs by aligning attention patterns.
Outcome:	The proposed method improves 13.56% (up to 30%) on the POPE and 21.75% on the hallucination subsets across languages.

Copied to clipboard

Weihong Zhong, Xiaocheng Feng, Liang Zhao, Qiming Li, Lei Huang, Yuxuan Gu, Weitao Ma, Yuan Xu, Bing Qin

MoLoRAG: Bootstrapping Document Understanding via Multi-modal Logic-aware Retrieval (2025.emnlp-main)

Challenge:	Large Vision-Language Models (LVLMs) suffer from multimodal hallucinations . however, the generated hallucines could influence the models’ subsequent generation .
Approach:	They propose a framework to evaluate LVLMs' behaviors when encountering generated hallucinations and a method to revise the output distribution of LVLs with the one derived from the residual visual input.
Outcome:	The proposed framework reduces the performance of open-source LVLMs by 31%, indicating that they are prone to accept the generated hallucinations and make false claims that they would not have supported without distractions.

Copied to clipboard

Xixi Wu, Yanchao Tan, Nan Hou, Ruiyang Zhang, Hong Cheng

SHARP: Steering Hallucination in LVLMs via Representation Engineering (2025.emnlp-main)

Challenge:	Document Understanding is a foundational AI capability with broad applications . Large Vision-Language Models (LLMs) can't handle multi-page document comprehension . a logic-aware retrieval framework for multi-modal, multi- page document understanding is proposed .
Approach:	They propose a logic-aware retrieval framework for multi-modal, multi-page document understanding . MoLoRAG uses semantic and logical relevance to deliver more accurate retrieval .
Outcome:	The proposed framework improves on four DocQA datasets and demonstrates 9.68% accuracy improvement over existing methods.

Copied to clipboard

Junfei Wu, Yue Ding, Guofan Liu, Tianze Xia, Ziyue Huang, Dianbo Sui, Qiang Liu, Shu Wu, Liang Wang, Tieniu Tan

Mitigating Hallucinations in Large Vision-Language Models by Self-Injecting Hallucinations (2025.findings-emnlp)

Challenge:	Large Vision-Language Models (LVLMs) generate responses that are plausible but incorrect or unsupported—commonly referred to as hallucinations.
Approach:	They propose a representation-level intervention framework that modulates hallucination-related features during inference by probing their encoded features.
Outcome:	The proposed framework reduces hallucinations while maintaining the performance and generalization capabilities of Large Vision-Language Models (LVLMs).

Copied to clipboard

Yifan Lu, Ziqi Zhang, Chunfeng Yuan, Jun Gao, Congxuan Zhang, Xiaojuan Qi, Bing Li, Weiming Hu

Zero-Shot Defense Against Toxic Images via Inherent Multimodal Alignment in LVLMs (2025.findings-emnlp)

Challenge:	Existing methods for hallucination mitigation are based on external dependency and require external annotations or auxiliary models for preference data collection.
Approach:	a new method is proposed to help model-generated hallucinations without external dependencies.
Outcome:	a new method that self-injects hallucinations into a generated response improves halluuutations mitigation.

Copied to clipboard

Wei Zhao, Zhe Li, Yige Li, Jun Sun

UI-E2I-Synth: Advancing GUI Grounding with Large-Scale Instruction Synthesis (2025.findings-acl)

Challenge:	Existing safeguards relying on pre-filtering or fine-tuning are costly and diminish overall utility.
Approach:	They propose a lightweight method that leverages LVLMs’ inherent multimodal alignment for zero-shot toxic image detection.
Outcome:	The proposed method achieves a 66.9% defense success rate with only 3.2% false positive rate and 7.2% overhead.

Copied to clipboard

Xinyi Liu, Xiaoyi Zhang, Ziyun Zhang, Yan Lu

Doc-PP: Document Policy Preservation Benchmark for Large Vision-Language Models (2026.findings-acl)

Challenge:	Graphical User Interface (GUI) agents that utilize human-like vision perception capabilities are gaining a wider applicability compared to GUI metadata-based approaches.
Approach:	They propose a large-scale data synthesis pipeline for generating varying complex instruction datasets using GPT-4o instead of human annotators.
Outcome:	The proposed model achieves superior performance in GUI instruction grounding, demonstrating the advancements of proposed data synthesis pipeline.

Copied to clipboard

Haeun Jang, Hwan Chang, Hwanhee Lee

Seeing No Evil: Blinding Large Vision-Language Models to Safety Instructions via Adversarial Attention Hijacking (2026.acl-long)

Challenge:	Existing safety research focuses on implicit social norms or text-only settings, overlooking the complexities of multimodal documents.
Approach:	They propose a benchmark to assess the safety of large vision-Language Models (LVLMs) they propose 'Document Policy Preservation Benchmark' to assess document policy compliance.
Outcome:	The proposed framework outperforms standard prompting defenses in the evaluation of multimodal documents.

Copied to clipboard

Jingru Li, Wei Ren, Tianqing Zhu

Rethinking Post-Unlearning Behavior of Large Vision-Language Models (2026.findings-acl)

Challenge:	Existing attacks optimize image perturbations to maximize harmful output likelihood, but suffer from slow convergence due to gradient conflict between adversarial objectives and the model’s safety-retrieval mechanism.
Approach:	They propose a push-pull approach which suppresses attention to system-prompt tokens and anchors generation on adversarial image features to avoid collisions.
Outcome:	The proposed approach reduces gradient conflict by 45% and achieves 94.4% attack success rate on Qwen-VL (vs. 68.8% baseline) with 40% fewer iterations.

Copied to clipboard

Minsung Kim, Nakyeong Yang, Kyomin Jung

MRFD: Multi-Region Fusion Decoding with Self-Consistency for Mitigating Hallucinations in LVLMs (2025.findings-emnlp)

Challenge:	Existing methods to remove knowledge from large vision-Language Models often fail to provide quality and informative post-unlearning responses.
Approach:	They propose a task that requires models to provide privacy-preserving yet informative responses for LVLMs.
Outcome:	The proposed method reduces the risk of unlearning after naive suppression by providing informative and visually grounded responses.

Copied to clipboard

Haonan Ge, Yiwei Wang, Ming-Hsuan Yang, Yujun Cai

MAGIC-VQA: Multimodal And Grounded Inference with Commonsense Knowledge for Visual Question Answering (2025.findings-acl)

Challenge:	Large Vision-Language Models often produce hallucinations due to the limited ability to verify information in different regions of the image.
Approach:	a new decoding method improves factual grounding by modeling inter-region consistency . the method identifies salient regions using cross-attention and generates initial responses for each .
Outcome:	a training-free decoding method reduces hallucinations and improves response consistency . the proposed method generates initial responses for each region and weights reliability weights among responses .

Copied to clipboard

Shuo Yang, Caren Han, Siwen Luo, Eduard Hovy

PATIMT-Bench: A Multi-Scenario Benchmark for Position-Aware Text Image Machine Translation in Large Vision-Language Models (2025.findings-emnlp)

Challenge:	Existing Large Vision-Language Models (LVLMs) lack integrated commonsense knowledge . lack of integrated common knowledge limits their robustness and accuracy in VQA .
Approach:	They propose a framework to enhance multimodal inference by integrating commonsense reasoning.
Outcome:	MAGIC-VQA improves comprehensive benchmark datasets, surpassing existing models in tasks requiring advanced commonsense reasoning.

Copied to clipboard

Wanru Zhuang, Wenbo Li, Zhibin Lan, Xu Han, Peng Li, Jinsong Su

Multilinguality

Challenge:	Current TIMT studies focus on providing translations for all text within an image, neglecting to provide bounding boxes and covering limited scenarios.
Approach:	They extend traditional TIMT into position-aware TIMt to support fine-grained translation . they introduce an Adaptive Image OCR Refinement Pipeline to refine results .
Outcome:	The proposed model supports fine-grained and layout-preserving translation . the experimental data highlight the scalability and generalizability of the model.

Diving into Mitigating Hallucinations from a Vision Perspective for Large Vision-Language Models (2025.findings-emnlp)

Copied to clipboard

Weihang Wang, Xinhao Li, Ziyue Wang, Yan Pang, Jielei Zhang, Peiyi Li, Qiang Zhang, Longwen Gao

Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decoding (2024.findings-acl)

Challenge:	Existing benchmarks focus on coarse-grained hallucination detection and fail to capture hallucinics . vision encoders exhibit unique hallucinian characteristics, but suboptimal of simple feature fusion.
Approach:	They propose a visual encoder that employs different training paradigms to instill inductive biases in visual encoded models.
Outcome:	The proposed system reduces hallucinations and improves model performance.

Copied to clipboard

Xintong Wang, Jingheng Pan, Liang Ding, Chris Biemann

V-ALPHASOCIAL: Benchmark and Self-Reflective Chain-of-Thought Generation for Visual Social Commonsense Reasoning (2025.findings-acl)

Challenge:	Recent research in large vision-language models has shown promising results, but the issue of hallucination remains.
Approach:	They propose an instruction-based method to reduce hallucinations in large vision-language models . they use disturbance instructions to exacerbate hallucinosity in multimodal fusion modules .
Outcome:	The proposed method reduces hallucinations in multimodal fusion modules by reducing alignment uncertainty and subtracting hallucines from the original distribution.

Copied to clipboard

Zongyu Lin, Zhikun Xu, Xiaohan Song, Yixin Wan, Xingcheng Yao, Tsung-Han Lin, Selina Song, Pranav Subbaraman, Ben Zhou, Kai-Wei Chang, Yizhou Sun

Rethinking Jailbreak Detection of Large Vision Language Models with Representational Contrastive Scoring (2026.acl-long)

Challenge:	Social commonsense reasoning is a multimodal task that requires both textual and visual cues.
Approach:	They propose a method that integrates visual cues into social commonsense reasoning tasks.
Outcome:	The proposed method improves social commonsense reasoning on a multimodal foundation model.

Copied to clipboard

Peichun Hua, Hao Li, Shanghao Shi, Zhiyuan Yu, Ning Zhang

Game on Tree: Visual Hallucination Mitigation via Coarse-to-Fine View Tree and Game Theory (2024.emnlp-main)

Challenge:	Large Vision-Language Models (LVLMs) are vulnerable to a growing array of multimodal jailbreak attacks, necessitating a generalizable defense that is efficient for practical deployment.
Approach:	They propose a framework that uses a lightweight projection to separate benign and malicious inputs in safety-critical layers.
Outcome:	The proposed framework enables a simple yet powerful contrastive score that differentiates true malicious intent from mere distribution shift.

Copied to clipboard

Xianwei Zhuang, Zhihong Zhu, Zhanpeng Chen, Yuxin Xie, Liming Liang, Yuexian Zou

Benchmarking Large Vision-Language Models on CFMME: A Comprehensive Chinese Financial Multimodal Evaluation Dataset (2026.acl-long)

Challenge:	Large vision-language models produce unfaithful visual hallucinations, also known as visual halluinations, which hinders their application in multimodal understanding and decision-making.
Approach:	They propose a plug-and-play train-free decoding algorithm for mitigating visual hallucinations . they leverage visual information to construct a coarse-to-fine visual view tree .
Outcome:	The proposed algorithm reduces visual hallucinations (VH) by leveraging visual information to construct a coarse-to-fine visual view tree (CFTree)

Copied to clipboard

Qian Chen, Xianyin Zhang, Yanzhi Liu, Lifan Guo, Feng Chen, Chi Zhang

Investigating and Mitigating Object Hallucinations in Pretrained Vision-Language (CLIP) Models (2024.emnlp-main)

Challenge:	Large Vision-Language Models (LVLMs) have expanded capabilities beyond text understanding . a novel Chinese financial multimodal evaluation benchmark is used to evaluate LVLM capabilities .
Approach:	They propose a Chinese financial multimodal evaluation benchmark to evaluate LVLMs' capabilities . the model has an overall accuracy of 66.11% and an average score of 77.18 .
Outcome:	The proposed model achieves an overall accuracy of 66.11% on the question answering task and an average score of 77.18 on detection, recognition, and information extraction tasks.

Copied to clipboard

Yufang Liu, Tao Ji, Changzhi Sun, Yuanbin Wu, Aimin Zhou

VisuoThink: Empowering LVLM Reasoning with Multimodal Tree Search (2025.acl-long)

Challenge:	Existing studies have revealed that Large Vision-Language Models suffer from hallucinations in practice, including object hallucines, spatial hallucinos, attribute hallucinications, etc.
Approach:	They propose to use CLIP model to mitigate object hallucinations by using a data augmentation method to create negative samples with a variety of hallucinian issues.
Outcome:	The proposed method mitigates object hallucinations and can be used as a visual encoder, effectively alleviating the object halluination issue in LVLMs.

Copied to clipboard

Yikun Wang, Siyin Wang, Qinyuan Cheng, Zhaoye Fei, Liang Ding, Qipeng Guo, Dacheng Tao, Xipeng Qiu

Efficient Inference for Large Vision-Language Models: Bottlenecks, Techniques, and Prospects (2026.findings-acl)

Challenge:	Existing approaches to large vision-language models fail to capture interleaved nature of human visual-verbal reasoning processes.
Approach:	They propose a framework that integrates visuospatial and linguistic domains to facilitate multimodal slow thinking by enabling progressive visual-textual reasoning.
Outcome:	Experiments show that VisuoThink significantly improves reasoning capabilities even without fine-tuning.

Copied to clipboard

Jun Zhang, Yicheng Ji, Feiyang Ren, Yihang Li, Bowen Zeng, Zonghao Chen, Ke Chen, Lidan Shou, Gang Chen, Huan Li

DCP: Dual-Cue Pruning for Efficient Large Vision-Language Models (2025.emnlp-main)

Challenge:	Large Vision-Language Models are hindered by a systemic efficiency barrier known as visual token dominance.
Approach:	They propose a systematic taxonomy of efficiency techniques structured around the inference lifecycle . they examine visual encoding, prefilling, and decoding to understand bottlenecks .
Outcome:	The proposed techniques reveal how upstream decisions dictate downstream bottlenecks . the proposed techniques include hybrid compression and modality-aware decoding .

Copied to clipboard

Lei Jiang, Zixun Zhang, Yuting Zeng, Chunzhao Xie, Tongxuan Liu, Zhen Li, Lechao Cheng, Xiaohua Xu

Do LVLMs Know What They Know? A Systematic Study of Knowledge Boundary Perception in LVLMs (2025.findings-emnlp)

Challenge:	Existing pruning methods for large vision language models use visual tokens to prune . existing methods fail to balance efficiency and semantic alignment due to large number of visual token.
Approach:	They propose a cross-modal pruning framework that considers textual semantics and visual self-attention to combine them to achieve efficient inference acceleration.
Outcome:	The proposed pruning framework can retain only 25% of the visual tokens, with a minimal performance degradation of only 0.063% on LLaVA-1.5-13B.

Copied to clipboard

Zhikai Ding, Shiyu Ni, Keping Bi

Natural Language Interfaces

Challenge:	Large Vision-Language Models (LVLMs) demonstrate strong visual question answering (VQA) capabilities but are shown to hallucinate.
Approach:	They propose three confidence-based methods to enhance LVLMs' perception . they propose probabilistic and consistency-based signals are more reliable indicators .
Outcome:	Experiments on three LVLMs across three VQA datasets show that LVLs possess a reasonable perception level but there is room for improvement.

Unveiling the Response of Large Vision-Language Models to Visually Absent Tokens (2025.emnlp-main)

Copied to clipboard

Sohee Kim, Soohyun Ryu, Joonhyung Park, Eunho Yang

Self-Training Large Language and Vision Assistant for Medical Question Answering (2024.emnlp-main)

Challenge:	Large Vision-Language Models (LVLMs) generate contextually relevant responses by jointly interpreting visual and textual inputs.
Approach:	They propose a method to classify whether an input token is visually grounded by reinterpreting question prompts or replacing the detected absent tokens during generation.
Outcome:	The proposed method mitigates the models’ tendency to falsely presume the visual presence of text input and its generality across various LVLMs.

Copied to clipboard

Guohao Sun, Can Qin, Huazhu Fu, Linwei Wang, Zhiqiang Tao

CoViPAL: Layer-wise Contextualized Visual Token Pruning for Large Vision-Language Models (2025.findings-emnlp)

Challenge:	Existing methods for collecting medical data are expensive and time-consuming.
Approach:	They propose a method to train a large-scale LVLM capable of auto-generating medical visual instruction data to improve data efficiency.
Outcome:	The proposed method shows that it performs well across three major visual question answering (VQA) benchmarks.

Copied to clipboard

Zicong Tang, Ziyang Ma, Suqing Wang, Zuchao Li, Lefei Zhang, Hai Zhao, Yun Li, Qianren Wang

Unlocking Multilingual Reasoning Capability of LLMs and LVLMs through Representation Engineering (2026.acl-long)

Challenge:	Existing methods to prune redundant vision tokens struggle in shallow layers due to the lack of contextual information.
Approach:	They propose a layer-wise contextualized visual token pruning method that uses a plug-and-play Pruning Module to prune redundant vision tokens.
Outcome:	The proposed method outperforms training-free pruning methods under equal token budgets and surpasses training based methods with comparable supervision.

Copied to clipboard

Qiming Li, Xiaocheng Feng, Yixuan Ma, Ruihan Chen, Zihe Tong, Zekai Ye, Xiachong Feng, Libo Qin, Haoyu Ren, Kun Chen, Yunfei Lu, Dandan Tu, Bing Qin

Mitigating Action-Relation Hallucinations in LVLMs via Relation-aware Visual Enhancement (2026.acl-long)

Challenge:	Existing approaches to enhance multilingual reasoning capabilities rely on costly multilingual training or employ prompting with external translation tools.
Approach:	They propose a training-free inference-time method to enhance multilingual reasoning capabilities via Representation Engineering without additional training data or tools.
Outcome:	The proposed method outperforms existing methods on four reasoning benchmarks in English and Thai and Swahili.

Copied to clipboard

Zhenxin Qin, Qiang Li, Qingzhuo Wang, Ruiyang Qin, Zhihua Wei, Wen Shen

How Jailbreak Defenses Work and Ensemble? A Mechanistic Investigation (2025.findings-emnlp)

Challenge:	Existing research has focused on mitigating object hallucinations but often overlooks more complex relation hallucines, especially action relations involving interactions between objects.
Approach:	They propose a framework to locate action-relevant image regions and enhance the LVLM’s attention to those regions by using a Relation-aware Visual Enhancement method.
Outcome:	The proposed method achieves superior performance in mitigating action-relation hallucinations with negligible additional inference cost.

Copied to clipboard

Zhuohan Long, Siyuan Wang, Shujun Liu, Yuhang Lai

ReCon: Active Defense against Large Vision-Language Model Jailbreaks via Reverse Safety Concept Injection (2026.findings-acl)

Challenge:	Jailbreak attacks, where harmful prompts bypass generative models’ built-in safety, raise serious concerns about model vulnerability.
Approach:	They propose to reframe the standard generation task as a binary classification problem to assess model refusal tendencies for both harmful and benign queries.
Outcome:	The proposed defenses improve model safety or optimize the trade-off between safety and helpfulness.

Copied to clipboard

Zheng He, Yiwei Wang, Hongxing Wang, Yujun Cai

Just KIDDIN’ : Knowledge Infusion and Distillation for Detection of INdecent Memes (2025.findings-acl)

Challenge:	Existing defense strategies neglect visual threats and lack of fine-grained specificity regarding specific attack semantics.
Approach:	They propose a black-box defense framework that maps unsafe concepts to fine-grained, constructive Safe Concepts.
Outcome:	a new black-box defense framework enhances robustness against jailbreak attacks . it maps detected unsafe concepts to fine-grained, constructive Safe Concepts . the proposed framework is available for free at http://www.epa.org/recon/ .

Copied to clipboard

Rahul Garg, Trilok Padhi, Hemang Jain, Ugur Kursuncu, Ponnurangam Kumaraguru

EffiVLM-BENCH: A Comprehensive Benchmark for Evaluating Training-Free Acceleration in Large Vision-Language Models (2025.acl-long)

Challenge:	Existing approaches to detect toxicity in online multimodal environments require common-sense reasoning and contextual awareness.
Approach:	They propose a hybrid neurosymbolic framework that unifies distillation of implicit contextual knowledge from Large Vision-Language Models and infusion of explicit relational semantics through sub-graphs from Knowledge Graphs.
Outcome:	The proposed framework outperforms state-of-the-art models on two datasets with improvements of 0.5%, and 10.6% in HatefulMemes Benchmark.

Copied to clipboard

Zekun Wang, MingHua Ma, Zexin Wang, Rongchuan Mu, Liping Shan, Ming Liu, Bing Qin

Zero-Shot Fine-Grained Image Classification Using Large Vision-Language Models (2025.findings-emnlp)

Challenge:	Existing methods for accelerating Large Vision-Language Models lack comprehensive evaluation across diverse backbones, benchmarks, and metrics.
Approach:	They propose EffiVLM-BENCH framework for evaluating absolute performance and generalization and loyalty.
Outcome:	The proposed framework offers insights into optimal strategies for accelerating LVLMs.

Copied to clipboard

Md. Atabuzzaman, Andrew Zhang, Chris Thomas

Can Hallucination Correction Improve Video-Language Alignment? (2025.findings-acl)

Challenge:	Large Vision-Language Models have demonstrated impressive performance on vision-language reasoning tasks, but their potential for zero-shot fine-grained image classification remains underexplored.
Approach:	They propose a method that transforms zero-shot fine-grained image classification into a visual question-answering framework.
Outcome:	The proposed method outperforms the current state-of-the-art approach and outperformed existing methods.

Copied to clipboard

Lingjun Zhao, Mingyang Xie, Paola Cascante-Bonilla, Hal Daumé Iii, Kwonjoon Lee

VAUQ: Vision-Aware Uncertainty Quantification for LVLM Self-Evaluation (2026.findings-acl)

Challenge:	Existing work on hallucination correction for large vision-language models focuses on mitigating hallucisations, but a new approach is needed to improve video-language alignment.
Approach:	They propose a self-training framework learning to correct hallucinations in descriptions that do not align with the video content.
Outcome:	The proposed framework improves video-language alignment by identifying and correcting inconsistencies in descriptions that do not align with the video content.

Copied to clipboard

Seongheon Park, Changdae Oh, Hyeong Kyu Choi, Sean Du, Sharon Li

Mitigating Hallucination in Large Vision-Language Models through Aligning Attention Distribution to Information Flow (2025.findings-emnlp)

Challenge:	Existing self-evaluation methods rely on a model’s ability to estimate the correctness of its own outputs, but they depend heavily on language priors and are therefore ill-suited for evaluating vision-conditioned predictions.
Approach:	They propose a vision-aware uncertainty quantification framework that measures how strongly a model’s output depends on visual evidence.
Outcome:	The proposed framework outperforms existing methods across multiple datasets.

Copied to clipboard

Jianfei Zhao, Feng Zhang, Xin Sun, Chong Feng

3MDBench: Medical Multimodal Multi-agent Dialogue Benchmark (2025.emnlp-main)

Challenge:	Decode-Only models propagate information from left to right, but the model's attention still focuses on the visual representations, resulting in hallucinations.
Approach:	They propose to leverage the core information embedded in semantic representations to enhance the model's visual understanding by leveraging the attention distributions.
Outcome:	The proposed method reduces hallucinations by 80% by aligning the attention distribution with the actual information flow.

Copied to clipboard

Ivan Sviridov, Amina Miftakhova, Tereshchenko Artemiy Vladimirovich, Galina Zubkova, Pavel Blinov, Andrey Savchenko

Dialogue Systems & Conversational Agents

Challenge:	Large Vision-Language Models (LVLMs) are being explored in medicine but their ability to conduct complex real-world telemedicine consultations remains underexplored.
Approach:	They propose to use large vision-language models to conduct telemedicine consultations using a framework that simulates patient variability and evaluates diagnostic accuracy and dialogue quality via Assessor Agent.
Outcome:	The proposed framework compares diagnostic strategies for open and closed-source LVLMs and shows that multimodal dialogue improves F1 score by 6.5% over non-dialogue settings.

Enhancing Large Vision-Language Models with Ultra-Detailed Image Caption Generation (2025.emnlp-main)

Copied to clipboard

Yu Zeng, Yukun Qi, Yiming Zhao, Xikun Bao, Lin Chen, Zehui Chen, Shiting Huang, Jie Zhao, Feng Zhao

CAPA: Contribution-Aware Pruning and FFN Approximation for Efficient Large Vision-Language Models (2026.findings-acl)

Challenge:	Existing pipelines for generating high-quality, ultra-detailed image captions are limited by the scarcity of image caption data.
Approach:	They propose a pipeline for generating high-quality, ultra-detailed image captions that integrates both pre-processing and post-processor stages.
Outcome:	The proposed pipeline improves LVLMs' perception and cognitive abilities across multiple vision-language benchmarks.

Copied to clipboard

Samyak Jha, Junho Kim

Symmetrical Visual Contrastive Optimization: Aligning Vision-Language Models with Minimal Contrastive Images (2025.acl-long)

Challenge:	Efficient inference in Large Vision Language Models is constrained by the high cost of processing thousands of visual tokens.
Approach:	They propose a framework that prunes visual tokens using attention contribution at critical functional transitions and reduces computations using efficient linear approximations.
Outcome:	The proposed framework achieves competent efficiency–performance trade-offs with improved robustness.

Copied to clipboard

Shengguang Wu, Fan-Yun Sun, Kaiyue Wen, Nick Haber

Benchmarking and Improving Large Vision-Language Models for Fundamental Visual Graph Understanding and Reasoning (2025.acl-long)

Challenge:	Recent studies have shown that Large Vision-Language Models (VLMs) tend to neglect image content and over-rely on language-model priors, resulting in errors in visually grounded tasks and hallucinations.
Approach:	They propose a novel finetuning objective that steers the model toward capturing important visual details and aligning them with corresponding text tokens.
Outcome:	The proposed method achieves up to 22% reduction in hallucinations and significant gains in vision-centric and general tasks while maintaining or improving the model's general abilities.

Copied to clipboard

Yingjie Zhu, Xuefeng Bai, Kehai Chen, Yang Xiang, Jun Yu, Min Zhang

Activating Distributed Visual Region within LLMs for Efficient and Effective Vision-Language Training and Inference (2025.acl-long)

Challenge:	Existing methods for enhancing understanding and reasoning abilities in graphbased tasks focus on specific graph types or tasks, posing challenges in designing versatile systems suitable for various tasks and graphs across diverse domains.
Approach:	They propose a structure-aware fine-tuning framework to enhance LVLMs with structure learning abilities through three self-supervised learning tasks.
Outcome:	Extensive evaluations on 14 LVLMs reveal that LVLs are weak in basic graph understanding and reasoning tasks, particularly those concerning relational or structurally complex information.

Copied to clipboard

Siyuan Wang, Dianyi Wang, Chengxing Zhou, Zejun Li, Zhihao Fan, Xuanjing Huang, Zhongyu Wei

Diagnosing Spatial Consistency across Perspectives and Viewpoints in Large Vision-Language Models (2026.acl-long)

Challenge:	Existing Large Vision-Language Models (LVLMs) learn visual capacity through visual instruction tuning.
Approach:	They propose a method for LVLMs to be trained by selective layers tuning . they propose removing non-critical layers outside the visual region .
Outcome:	The proposed approach preserves nearly 99% of visual performance and improves textual task results while reducing training time.

Copied to clipboard

Yoonji Kim, Jieun Kim, Yujin Jeong, Sung-Bae Cho

PseudoGD: Enhancing Spatial Reasoning in Vision-Language Models through Pseudo Geometric Knowledge Distillation (2026.findings-acl)

Challenge:	Existing models assess spatial capabilities from a static, single-view and egocentric perspective, failing to capture the dynamic nature of real-world spatial cognition.
Approach:	They propose a benchmark to diagnose spatial reasoning capabilities using a 360 field of view.
Outcome:	The proposed benchmark evaluates allocentric and egocentric reasoning capabilities from multiple perspectives in high-quality 3D environments.

Copied to clipboard

Gwanghee Lee, Yeeun Choi, Kyoungson Jhang

When Big Models Train Small Ones: Label-Free Model Parity Alignment for Efficient Visual Question Answering using Small VLMs (2025.emnlp-main)

Challenge:	Recent Large Vision-Language Models (LVLMs) have shown remarkable success in general semantic understanding, but struggle with 3D spatial reasoning tasks.
Approach:	They propose a framework to help vision encoders internalize 3D geometric information using only standard 2D images.
Outcome:	The proposed framework achieves State-of-the-Art (SOTA) performance across various model architectures.

Copied to clipboard

Abhirama Subramanyam Penamakuri, Navlika Singh, Piyush Arora, Anand Mishra

Before Forgetting, Learn to Remember: Revisiting Foundational Learning Failures in LVLM Unlearning Benchmarks (2026.findings-acl)

Challenge:	Large vision and language models have demonstrated remarkable performance in visual question answering tasks.
Approach:	They introduce a framework to optimize L-VLMs by leveraging unlabeled images . they conduct extensive experiments on four diverse VQA benchmarks .
Outcome:	The proposed framework improves L-VLMs on four visual question answering benchmarks.

Copied to clipboard

Junehyoung Kwon, MiHyeon Kim, Eunju Lee, JungMin Yun, Byeonggeuk Lim, YoungBin Kim

Challenge:	Large Vision-Language Models (LVLMs) are capable of learning from vast webscale datasets but pose privacy risks as they can unintentionally memorize sensitive information.
Approach:	They propose a Reliable Multi-hop and Multi-image Memorization Benchmark that ensures robust foundational learning through principled data scaling and reasoning-aware QA pairs.
Outcome:	Extensive experiments show that ReMem provides a reliable framework for diagnosing both learning and unlearning behaviors in Large Vision-Language Models.

Benchmarking and Mitigating MCQA Selection Bias of Large Vision-Language Models (2025.emnlp-main)

Copied to clipboard

Md. Atabuzzaman, Ali Asgarov, Chris Thomas

DiVE: Decoupling Intra-layer Visual Evidence for Mitigating Hallucinations in Large Vision-Language Models (2026.acl-long)

Challenge:	Existing work has explored unimodal biases in visual question answering, but the problem of selection bias in Multiple-Choice Question Answering (MCQA) remains underexplored.
Approach:	They propose a method that mitigates bias without retraining and is compatible with frozen LVLMs.
Outcome:	The proposed method mitigates bias without retraining and is compatible with frozen LVLMs.

Copied to clipboard

Xinwei Li, Li Lin, Hui Jiao, Li Yao, Tien-Tsin Wong, Hanqian Wu

Vocabulary Hijacking in LVLMs: Unveiling Critical Attention Heads by Excluding Inert Tokens to Mitigate Hallucination (2026.acl-long)

Challenge:	Existing decoding-based approaches do not explicitly decouple visual evidence from mixed vision–language representations.
Approach:	They propose to decouple visual evidence from mixed vision–language representations by dynamically identifying layers enriched with visual information and performing intra-layer decoupling to extract aggregated visual evidence.
Outcome:	Experiments show that DiVE achieves state-of-the-art performance on multiple benchmarks.

Copied to clipboard

Yangneng Chen, Junlin Li, Weijun Yao, Xilai Ma, Guodong DU, Wenya Wang, Jing Li

Aligning with Your Own Voice: Self-Corrected Preference Learning for Hallucination Mitigation in LVLMs (2026.findings-acl)

Challenge:	Large Vision-Language Models (LVLMs) are capable of processing visual inputs, but are susceptible to hallucinations.
Approach:	They propose a method to localize and localize specific visual tokens, which are defined as Inert Tokens, across layers, revealing a rigid semantic collapse.
Outcome:	The proposed approach reduces the likelihood of LVLMs being hijacked by visual inputs while maintaining general capabilities.

Copied to clipboard

Byeonggeuk Lim, JungMin Yun, Junehyoung Kwon, Kyeonghyun Kim, YoungBin Kim

Towards Mitigating Hallucinations in Large Vision-Language Models by Refining Textual Embeddings (2026.findings-acl)

Challenge:	Existing preference learning-based approaches rely on proprietary models to construct preference datasets, causing a distributional mismatch between the proprietary and target models.
Approach:	They propose a framework that aligns LVLMs using in-distribution data derived from the model's intrinsic knowledge.
Outcome:	The proposed framework surpasses baselines in hallucination mitigation while requiring only 5.2k samples.

Copied to clipboard

Aakriti Agrawal, Gouthaman KV, Rohith Aralikatti, Gauri Jagatap, Jiaxin Yuan, Sarvesh Baskar, Vijay Kamarshi, Andrea Fanelli, Furong Huang